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DESCRIPTION 

Method and system for control of a device. 

This invention relates in general to a method and system for control of a device and, in 
particular, to a method and system for remote control of a consmner electronics device, 
e.g. audio-visual device such as television (TV), video cassette recorder (VCR), digital 
video disc (DVD) player, personal computer (PC) etc. 

Remote controls are used today together with almost any consumer electronics device, 
e.g. television, DVD player, tuner, etc. In the average household, multiple remote 
controls - often one for each consumer electronics device - can be required. Even for a 
person well acquainted with the consumer electronics devices he owns, it is a challenge 
to remember what each button on each remote control is actually for. Furthermore, the 
on-screen menu-driven navigation available for some consumer electronics devices is 
often less than intuitive, particularly for users that might not possess an in-depth 
knowledge of the options available for the device. The result is that the user must 
continually examine the menu presented on the screen to locate the option he is looking 
for, and then look down at the remote control to search for the apprqpriate button. Quite 
often the buttons are given non-intuitive names or abbreviations. Additionally, a button 
on the remote control might also perform a further function which is accessed by first 
pressing a mode button. The multitude of options available for mod^ consumer 
electronics devices unfortunately mean lhat for many users, programming such a device 
can become an exercise in frustration. The large number of buttons and non-intuitive 
menu options can make the programming of a device more difficult than necessary and 
often result in the user not getting the most out of the devices he has bought. 

Using all one's consumer electronics devices to the fidl is made even more difficult by 
the feet that almost every consumer electronics device today comes with its own remote 
control device. Whilst most remote control button abbreviations and symbols are by 
now standardised to aUow marketing of the same remote control device m counties of 
different languages, even so it might be tiiat different abbreviations or symbols are used 
on different remote controls to perform the same fimction, for example the abbreviation 
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"CH" and "PR" might be used to indicate "channel" or "program", meaning essentially 
the same thing. The remote controls also differ in shape, size, overall appearance and 
even battery requirements. 

5 In an effort to reduce the confusion caused by such a multitude of remote controls, a 
new product category of "universal remote controls" has been developed. However, 
even a universal remote control cannot hope to access all the ftmctions offered by every 
consumer electronics device avmlable on the market today, particularly since new 
technolo^es and features are continually being developed. Furthermore, the wide 
1 0 variety of functions offered by modem consumer electronics devices necessitates a 
correspondingly large number of buttons to invoke these functions, requiring an 
inconveniently large remote control to accommodate all the buttons. 

A number of developments attempt to address the problem of simple remote control of 
1 5 consumer electronics devices. For example EP 0372674 suggests a method involving a 
camera and a stationary light source, both connected to a microcomputer. The moveable 
camera, containing a lens and a detector, is mounted on the head of the user. 
Movements of the user's head relative to the fixed light source, detected by the detector 
in the camera, are translated into movements of a cursor on a television screen. An 
20 obvious disadvantage of such a method is that the user has to wear a camera on his 
head. He must move his head in various directions in order to move the cursor on the 
screen, whilst continually watching the screen to see if he is directing the cursor to the 
desired position. 

25 Therefore, an object of the present invention is to provide a metihod and user interface 
for convenient and intuitive remote control of a device. 

To this end, the present invention provides a method for control of a device, which 
method comprises visually presentmg a number of user options for the device to be 
30 controDed, aiming a pointing device conqmsing a camera at the visual presentation of 
the user options to choose a desned option, generating an image of a target area aimed 
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at by the pointing device, and comparing the target area image with a pre-defined 
template of the visual presentation to determine the chosen option. The phrase 'target 
area image" is to be understood in the broadest possible sense, for example the target 
area image might comprise merely image date concerning significant points of the 
entire image, e.g. enhanced contours, comers, edges etc. 

An ^propriate user interface for remote control of a device comprises an accessing unit 
for accessing pre-defined templates associated with visual presentations of user options 
for the device to be controlled, a pointing device for aiming at a desired option in a 
visual presentation of the user options, comprising a camera for generating an image of 
a target area of at least part of the visual presentation, and an image interpreter for 
locating the target area or a point of the target area in a pre-defined template in order to 
determine the chosen option. 

The method and the user interface thus provide a comfortable way for a user to control 
any device by simply aiming a compact hand-held pointing device to make an option 
selection without having to look away from the consumer electronics device while 
making a selection, and without first having to become acquainted with a multitude of 
buttons. 

The dependent claims and the subsequent description disclose particularly 
advantageous embodiments and features of the invention. 

User options for a consumer electronics device can be presented to the user in a number 
of ways, both stetic and dynamic. The simplest visual presentation of the user options 
for a device in static form is the firont of the device itself, where various options are 
available in the fomi of buttons or knobs, for example the stop, fast forward, record and 
play buttons on a VCR. Another example of a static visual presentation might be to 
show the user options in printed form, for example as a computer printout, or a program 
guide in a TV magazine. Especially for a device such as a TV, or DVD player which 
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can be connected to a television, the options may be available to the user in static form 
as buttons on the front of the device, and can also easily be dynamically displayed on 
the television screen. Here, the options might be shown in the form of menu items or as 
icons. In a particularly preferred embodiment of the invention, user options for more 
5 than one device can be shown simultaneously in one visual presentation. For exainple, 
tuner options and DVD options might be displayed togethw, particularly options that 
are relevant to both devices. One example of such a combination of options might be to 
display a set of tuner audio options such as surround sound, Dolby etc, along with DVD 
options such as wide screen, sub-titles etc. The user can thus easily and quickly 
10 customise the options for both devices before commencing viewing. 

Another way of visually presenting the user options for a device in dynamic form might 
be to project them as an image backdrop onto a surface, for example a wall or a screen. 
Additionally, the invention might easily provide the user with a means of personalising 

15 the options for the device, for example by only displaying a small number of options on 
the screen at one time if the user has poor vision. Further, the user might specifically 
choose to omit functions that he is unlikely ever to require, for exanple, for his DVD 
player, he migiht never wish to view a film accompanied by foreign-language subtitles. 
In this case, he can personalise his user interface to omit these options from the visual 

20 presentation. A device such as a television can be configured so that for some users, 
only a subset of the available options is accessible. In this way, certain channels can be 
made accessible only by authorised users, for example to protect children from 
watching programs unsuitable to their age group. 

25 It is a central point of the invention that the user can choose among the options 

available by arming a pointing device containing a camera at the visual presentation of 
the user options. The camera is preferably incorporated m the pointing device but might 
equally be mounted on the pointing device, and is preferably oriented in such a way that 
it generates images of the area in front of the pointing device targeted by the user. The 

30 hnage of the target area might be only a small subset of the entire visual presentation, it 
might cover the visual presentation in its entirety, or it might also include an area 
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surrounding the visual presentation. The size of the target area image in relation to the 
entile visual presentation might depend on the size of the visual presentation, the 
distance between the pointing device and the presentation, and on the capabilities of the 
camera itself. The user might be positioned so that the pointing device is at some 
distance from the visual presentation, for example when the user is seated whilst 
watching television. Equally, the user might hold the pointing device quite close to the 
visual presentation, as might arise when the user is aiming the pointing device at a TV 
program guide in magazine form. 

In a preferred embodiment of the invention, a light source might be mounted in or on 
the pointing device. The light source might serve to illuminate the area at which the 
pointing device is aimed, so that the user can easily peruse the visual presentation even 
if the surroundings are dark. Equally, the light source might be a source of laser light 
source of a concentrated beam of light emitted in the direction of pointing, so that a 
point of light appears at or near the target point on the visual presentation at which the 
user is aiming, providing visual positional feedback to help the user aim at the desired 
option. A simple realisation might be a laser light source incorporated in or mounted on 
the pointing device in an ajjpropriate naanner. In the following therefore, it is assumed - 
without limiting the invention in any way - that the source of concentrated light is a 
laser beam. 

For ease of use, the pointing device can be in the shape of a wand or pen in an 
elongated form that can be grasped comfortably by the user. The user can thus direct 
the pointing device at a target point in the visual presentation while positioned at a 
comfortable viewing distance from it. Equally, the pointing device might be shaped in 
the form of a pistol. 

The pointing device might be aimed by the user at a particular option in a visual 
presentation, for example at the play button on the front of a VCR device, at a DVD 
option displayed on a TV screen, or at a selected program in a TV magazine. To 
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indicate that a particular selection is being made, the user might move the pointing 
device in a pre-defined manner over the visual presentation, for example by describing 
a loop or circular shape around the desired option. The user might move the poiating 
device through the air at a distance removed from visual presentation, or might move 
the pointing device directly over or very close to the visual presentation. Another way 
of indicating a particular option selection might be to aim the pointing device steadily at 
the option for a pre-defined length of time. Equally, the user might flick the pointing 
device across the visual presentation to indicate, for exawple, a return to normal 
program viewing after removing the visual presentation firom the screen, or to a 
previous menu level. The movement of the pointing device relative to the visual 
presentation might preferably be detected by image processing software in the control 
unit, or might be detected by a motion sensor in the pointing device. A further 
possibility might be to press a button on the pointing device to indicate selection of the 
option at which the pointing device is aimed. In a preferred embodiment, the control 
unit can initiate a confirmation dialog in order to ascertain that it has correctly 
interpreted the user's actions, for example if the user has aimed at a point considerably 
removed from the optical centre of an option while pressing the button or moving the 
pointing device in a pre-defined manner. In this case the contix>l unit might request 
confirmation before proceeding to initiate the selected option or function. 

The control unit can preferably alter the visual presentation to highli^t the selected 
option in some way, for example by making the option appear to flash or by 
highlighting the region in the visual presenlation aimed at by the user perhaps 
accompanying this by an audible "cUck" sound. The pomting device might also select a 
function in the visual presentation using a "drag and drop" technique, particularly when 
the user must navigate through larger content spaces for example by dragging an icon 
representing buffered DVD movie data to another icon representing a trash can, thus 
indicating that the buffered data be deleted from memory. Various functions might be 
initiated by the user, whereby flie user selects the option in a manner similar to a 
"double-click", for example, by repeating the motion of the pointing device in the pie- 
defined manner, or twice pressing a button on the pointing device. 
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In a particularly advantageous embodiment of the invention, the image of the target 
area is transmitted jfrom the pointing device in a wireless manner, for example using 
Bluetooth or 802.1 lb standards, to a control unit assigned to the device to be controlled. 
The control unit preferably comprises a receiver for receiving the target area images 
5 and an image interpreter, which might be in the form of a processor or controller. The 
target area images might equally be transmitted to the control unit by means of a cable 
attaching the pointing device to the control unit. 

To determine which option has been selected by the user, the image interpreter 
10 coBopares the received target area images to a number of pre-defined tenq)lates of the 
visual presentation. A single pre-defined ten:q>late might suffice for the con^arison, or 
it may be necessary to ^jply more than one teniplate in order to make a successful 
conparison. 

1 5 Pre-defined templates can be stored in an internal memory, or mi^t equally be 

accessed from an external source. Preferably, the control unit comprises an accessing 
unit with an appropriate interface for obtaining pre-defined templates for the visual 
presentation of the device to be controlled firom, for example, an internal or external 
memory, a memory stick, an intranet or the internet. A template can be a graphic 

20 representation of tibie firont of the device to be controlled, for example a shnplified 
representation of the firont of a VCR device featuring the user options available, for 
example the buttons representing the play, fast-forward, rewind, stop and record 
functions. A template can also be a gr^hic representation of an options menu as 
displayed on a TV screen and might indicate the locations of the available device 

25 options associated with particular areas of the visual presentation. For example, the user 
options for a DVD player such as play, fast-forward, sub-titles, language etc., can also 
be visually presented on the TV screen. The template can also depict the area around 
the visual presentation, for example it may include the housing of the device, and may 
even include some of the immediate surroundings of the device. 

30 

User options for a device which can display these on a screen can often be presented in 
the form of menus, where the user can traverse the menus to arrive at the desired option 
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or function. In aprefeired embodiment of the invention, a template exists for each 
possible menu level for the device to be controlled, so that the user can aim the pointing 
device at any one of the available options at any level of control of the device. Another 
tj^e of template mi^t have the appearance of a TV program guide in a magazine. 
5 Here, templates for the layout of the pages in the TV guide naight be obtained and/or 
updated by the accessing unit, for example on a daily or weekly basis. Preferably, the 
image interpretation software is compatible with the format of the TV guide pages. The 
templates preferably feature the positions on the pages of the various program options 
available to the user. The user might aim the pointing device over the visual 
1 0 presentation in the form of a page in an actual TV program guide to select a particular 
option, or the guide might be visually presented on the TV screen at which the user can 
aim the pointing device to choose between the options available. 

For processing the target area image in order to determine the chosen option, it is 
1 5 expedient to apply computer vision techniques to find the point in the visual 
presentation at which the user has aimed, i.e. the target point. 

Jn one embodiment of the invention, a iBxed point in the target area image, preferably 
the centre of the target ar^ image, obtained by extending an imaginary line in the 
20 direction of the longitudinal axis of the pointing device to the visual presentation, might 
be used as the target point. 

A method of processing the target area images of the visual presentation using 
computer vision algorithms might comprise detecting distinctive points in the target 

25 image and determining corresponding points in the template of the visual presentation, 
and developing a transformation for mapping the points in tbe target image to the 
corresponding points in the template. The distinctive points of the target area image 
might be points of the visual presentation, or mi^t equally be points in the area 
surrounding the visual presentation, for exansplG the comers of a television screen, or 

30 points belonging to an object in the vicinity of the device to be conti-oUed and which are 
also recorded in the pre-defined templates. This transformation can then be used to 
determine the position and aspect of the pointing device relative to the visual 
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presentation so that the intersection point of an axis of the pointing device with the 
visual presentation can be located in the template. The position of this intersection in 
the template corresponds to the target point on the visual presentation, and can be used 
to easily determine which of the options has been targeted by the user. The position of 
the target point in the pre-defined template indicates the option selected by the user- Ib 
this way, comparing the target area image with the pre-defined template is restricted to 
identifying and comparing only salient points such as distinctive comer points. The 
term "comparing" as applicable in this invention is to be understood in a broad sense, 
i.e. by only conq)aring sufficient features in order to quickly identify the point at which 
the user is aiming. 

Another possible way of determining the option selected by the user is to directly 
coirpare the received target area image, centred around the target point, with a pre- 
defined template to locate the point targeted in the visual presentation using methods 
such as pattem-matching. Another way of comparing the target area image with the pre- 
defined template restrict itself to identifying and comparing only salient points such as 
distinctive comer points. 

In a further embodiment of the invention, the location of the laser point, transmitted to 
the receiver in the control unit as part of the target area image, might be used as the 
target point to locate the option selected by the user. The laser point may be 
superimposed on the centre of the target area image, but might equally well be oiBFset 
j&om the centre of the target area image. 

The control unit can be used to control a single device, but might equally be used to 
control a plurality of such devices, for example one control unit might be assigned to a 
TV, a DVD player, a tuner and a VCR. In this way, the pointing device can be used as a 
remote control for one device or for a plurality of devices. The pointing device can also 
be used to control a number of consumer electronics devices, each incorporating their 
own dedicated control units. Furthermore, a number of pointing devices can be assigned 
to a control unit, so that, for example, each member of a household has his own 
pointing 
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device. On the other hand, one pointing device might be assigned to a number of 
control units in different environments, for example so that a user might use his 
pointing device for controlling devices at home as well as in the office. 

In a preferred embodiment of the invention, the control unit might be connected to a 
projector which can project visual presentations of user options for a number of devices 
in the form of an image backdrop onto a suitable surfece, for example a wall. The 
control unit might also avail of a separate screen, or might use a screen of one of the 
devices to be controlled. In this way, user options can be presented in a comfortable 
manner for a device which does not otherwise feature a display, for example a video 
recorder. The options represented by the buttons on the compact front of the device can, 
for example, be presented as menu options on the larger image backdrop for ease of 
selection. In a further preferred embodiment of the invention, the control unit can 
produce a hard-copy of a visual presentation, for example it can print out a program 
selection- 
Other objects and features of the present invention will become apparent from the 
following detailed descriptions considered in conjunction with the accompanying 
drawing. It is to be understood, however, that the drawings are designed solely for the 
pinposes of illustration and not as a definition of the limits of the invention. 

Fig. 1 is a schematic diagram showing a user with a pointing device and a device to be 
controlled; 

Fig. 2 is a schematic diagram of a pointmg device in accordance with an embodiment of 
the present invention; 

Fig. 3 is a schematic diagram of a visual pres^tation of a device to be controlled and 
an associated target area image; 

Fig. 4 is a schematic diagram of a device to be controlled comprising a control unit and 

a pointing device in accordance with an embodiment of the present invention; 

Fig. 5 is a schematic diagram of a control unit and a plurality of devices to be controlled 
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in accoidance with an embodiment of the present invaition; Fig. 6 is a schematic 
diagram showing a visual presentation and a correspondmg target area image in 
accordance with an embodiment of the present invention. 

Fig. 1 shows a user 8 aiming a pointing device 2 at a device la to be controlled, in this 
case a television la. In this embodiment, the visual presentation 4a of the user options is 
dynamically displayed on the television screen. The pointing device 2 generates images 
of areas targeted on the visual presentation 4. The television 1 contains a control unit 
which receives the images transmitted from the pointing device 2. 

Fig. 2 shows a pointing device 2 containing a camera 3, which generates images of the 
area m front of the pointing device 2 m the dkection of pomting D. The pointing device 
has 2 features an elongated form, so that the dhection of pointing D lies along the 
longitudinal axis of the pointmg device 2. The camera 3 is positioned towards the front 
of the pointing device 2 so that images are generated of the area in front of the pointing 
device 2 at which the user 8 is aiming. The images are transmitted by means of a 
transmitter 14 enclosed in the housing of the pointing device 2 and are transmitted in a 
wireless manner, e.g. Bluetooth, or 802.11b standards A laser light source 7, mounted 
on the pomting device 2, emits a beam of laser light L parallel to the longitudinal axis 
of the pointing device 2 in the direction of pomting D. In this embodiment, the pointing 
device 2 features a button 1 1 . The button 1 1 can be pressed by the user, for example to 
confirm that he has made a selection and to transmit the image of the target area. 
Altematively, such a button 1 1 might additionally be used to activate or deactivate 
displaying of the visual presentation, so that the user interfece options are only 
displayed when actually required by the user. Altematively, the function of the button 
1 1 might be to activate or deactivate the light source 7 mounted on the pointing device, 
or to activate or deactivate the pomtiog device 2 itself Equally, the pointing device 
might be activated by means of a motion sensor incorporated in the pointing device 2, 
so fliat the visual presentation is automatically displayed when the user takes hold of the 
pointing device 2. The pointing device 2 draws its power from one or more batteries, 
not shown in the figure. Depending on the consumption of the pointing device, it may 
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be necessary to provide a cradle into which the pointing device 2 can be placed when 
not in use, to recharge the batteries. 

Fig. 3 shows a visual presentation 4a - as displayed on a television screen or projected 
onto an appropriate backdrop - of a number of user options for a device to be 
continued. A pointing device, not shown in the jBgure, is being aimed at a target area 6 
of the visual presentation 4a. With the aid of a laser point Pl generated by a laser light 
source on the pointing device, the user can select one of the menu options (Mi, Ma, M3, 
M4) shown in the visual presentation 4a. A camera in the pointing device generates an 
image 5 of the target area 6 centred around an image cenfre point Pt. The laser point Pl 
also appears in the target area hnage 5. The laser point Pl may be a distance removed 
firom the image centre point Pt, or might coincide with the image centre point Pt. 

Fig. 4 shows a consumer electronics device 1 incorporating a display, for example a 
television screen, and a control unit 16. The control unit 16 comprises a receiver 17 
which is capable of receiving target area images 5 of a visual presentation 4a, shown in 
this case on a screen of the device 1, transmitted from a pointing device 2 aimed at the 
visual presentation 4a. The target area image data 1 8 is passed to an image interpreter 
13. An accessing unit 12 can access predefined tenyplates, for exanple from an internal 
memory 19, an external memory 20, or the internet 21. Ideally the accessing unit 12 has 
a number of interfeces allowing access to external data 20, for example the user might 
provide pre-defined templates stored on a memory medium 20 such as floppy disk, CD 
or DVD. The templates may also be configured by the user, for example in a training 
session for the control unit 16, in which the user specifies the correlation between 
specific areas on the template with particular functions. 

The image interpreter 13 makes use of computer vision algorithms to compare the 
target area hnages 1 8 to the predefined templates in order to determine the option 
selected by the user. Information about the chosen option is forwarded to a dialog 
controller 10, which acts to instruct the device 1 to carry out the function associated 
with the option, e.g. to switch to the selected channel, and also instixicts a visual 



-13- 



PHDE030423 EPP 



presentation controller 

9 wMch, for example, updates the visual presentation 4a to show a dijfferent menu level 
or deactivates the visual presentation 4a. 

Fig. 5 shows a number of consumer electronics devices la, Ic comiected to a control 
unit 16. The devices U, Ic feature different means of visually presenting to a user 
the options fhey proffer. Device la, for example a television, displays its user options on 
its own screen 4a- A user can aim a pointing device at the options displayed on the 
television screen 4a, for exanq3le to switch channels, change loudspeaker volume, view 
videotext etc. The fronts of the devices lb and Ic, for example a DVD player or VCR, 
display the user options for these devices and are used as visual presentations 4b and 4c- 

Here, the user can aim the pointing device at one of lb or Ic to, for example, fest- 
forward the DVD or video cassette contained in the device. 

Another visual presentation 4e is shown in the form of a TV program guide le. The user 
can aim a pointing device over the pages of the program guide 4e to select a desired 
program. The control unit 16 then acts accordingly, for example by issuing commands 
to the television la to switch to the appropriate channel if the program is already 
running. If the program conamences at a later point in time, the control unit 16 might 
display appropriate prompts on the television screen 4a, for example, whether the 
program is to be recorded later by the VCR To ensure compatibility between image 
interpretation software and TV guide format, the soflware can be updated on a regular 
basis or as required by downloading up-to-date versions from the internet 21 . 

A beamer or projector 1 5 displays a fiarther visual presentation 4d in the form of an 
image backdrop, which might be associated with one or more of la, lb, Ic with le, or 
with another device not shown in the figme, onto a wall or other suitable surface. 
Ltnages of target areas of any of the visual presentations 4e transmitted from a pointing 
device (not shown in the figure) are received by the control unit 16 and compared to 
pre-defined templates of the visual presentations 4a, 4b, 4c, 4d , 4e. The predefined 
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teii5)lates can be stored locally in the control unit. For example, predefined templates 

for devices lb and Ic can be stored once in the control unit, since the front of such a 
device remains static. Templates for the visual presentations for a device such as a 
5 television U might change intermittently, for example it might be possible for a user to 
program the device to customise the menu-driven operation, so that the menu levels and 
their corresponding templates might take on a different appearance. The templates for a 
TV program guide are iqwiated by the control unit with every issue of the program 
guide. These texapMes are downloaded from the internet 21 by means of a suitable 
1 0 interface. The control unit 1 6 determines which of the visual presentations 4a, 4b, 4c, 4d, 
4e has been targeted, and which option has been selected. The control unit then issues 
appropriate commands to the devices in order to carry out the requested fimction. 

Figure 6 shows a schematic representation of a target area image 5 generated by a 
1 5 pointing device 2 and a template for the visual presentation 4a. The pointing device is 
aimed at the visual presentation from a distance and at an oblique angle, so that the 
scale and perspective of the elements (M i, M2, M3) in the visual presentation 4a appear 
distorted in the target area image 5. To determine the option selected by the user, in this 
embodiment the point of intersection Pt of the longitudinal axis of the pointing device 2 
20 with the visual presentation 4^^ is located The point in the template corresponding to 
the point of intersection Pt can then be located to determine the chosen option. To this 
end, computer vision algorithms using edge- and comer detection methods are applied 
to locate points in the target area image [(xa, ya), (xb, yb), (xe, yc)] which correspond to 
points in the template [(xa', ya'), (xb', yb'), (x^', yc')] of the visual presentation 4a. 

25 

Each point can expressed as a vector e.g. the point (Xa, ya) can be expressed as v^ . As a 

next step, a transformation ftmction T^ is developed to map the target area image to the 
template: 

fW=Z|Tx(v.)-v|f 

i 

30 where the vector V; represents the coordinate pair (xi, yO in the target area image, and 
the vector y[ represents the corresponding coordinate pair (x'i, y'O in the template. The 
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parameter set X, comprising parameters for rotation and translation of the image 

yielding the most cost-effective solution to the ftinction, can be applied to determine the 
position and orientation of the pointing device 2 with respect to Ihe visual presentation 
4a. The computer vision algorithms make use of the fact that the camera 3 within the 
pointing device 2 is fixed and "looking" in the direction of the pointing gesture. The 
next step is to calculate the point of intersection of the longitudinal axis of the pointing 
device 2 in the direction of pomting D with the plane of the visual presentation 4a. This 
point may be taken to be the centre of the target area image Pt, or, if the device has a 
laser pointer, the laser point Pl can be used instead. Once the coordinates of the point of 
intersection have been calculated, it is a sunple matter to locate this point in the 
template of the visual presentation 4,, thus determining the option which the user has 
selected. 

Although the present invention has been disclosed in the form of preferred 
embodiments and variations thereon, it will be understood that numerous additional 
modifications and variations could be made thereto without departing flrom the scope of 
the invention. The remote control pointing device can serve as the universal user 
interface device in the home or for navigation through business presentations. In short, 
it can be beneficial whenever an intention of a user can be expressed by pointing, which 
means that it can be used for essentially any kind of user interface. Its small form factor 
and its convenient and intuitive usage can elevate such a sunple pointing device to a 
powerfiil universal remote control. Its ability to be used to control a multitude of 
devices, providing access to content items of the devices, as well as allowing for 
personalisation of the device's user mterface options, make this a powerful tool. As an 
alternative to the pen shape, the pointing device could for example also be a personal 
digital assistant (TPDA) with a built-in camera, or a mobile phone with a built-in camera. 
The pointing device might be combined with other traditional remote control features, 
e.g. with additional buttons for performing dedicated functions such as fest scrolling 
through a long list of items, or with other input modalities such as voice control for 
direct access to content items of the device to be controlled. 
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The usefulness of the pointing device need not be restricted to the control of consumer 
electronics devices, for exanaple it may equally find application within a medical 
environment, in a home or oflBce environment, or in industry. For example in a home 
environment, such a pointing device could be used to control almost any electrical 
5 appliance which can be controlled by a control unit, for example washing machine, 
cooker, heating, light fittings. Such a pointing device could make life considerably 
easier for users who are handicapped or so restricted in then: mobility that Ihey are 
unable to reach the appliances or to operate Ihem in the usual manner. 

10 For the sake of clarity, it is also to be understood that the use of "a" or "an" throughout 
this application does not exclude a plurality, and "comprising" does not exclude other 
steps or elements. A "unit" may comprise a number of blocks or devices, unless 
explicitly described as a single entity. 
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CLAIMS 



1 . A method for control of a device (la, U, Ic), which method comprises: 

- visually presenting a nmnber of user options for the device to be controlled 

(la, Ibj Ic)} 

- aiming a pointing device (2) comprising a camera (3) at the visual 

5 presentation (4a, 4b, 4o, 4d, 4e) of the user options to choose a desired option; 

- generating an image (5) of a target area (6) aimed at by the pointing device 
(2); 

- coir^aring the target area image (5) with a pre-defined template of the visual 
presentation (4a, 4b, 4c, 4d, 4e) to determine the chosen option. 

10 

2. A method according to claim 1, where a source of a concentrated beam of light (7) 
attached to the pointing device (2) shows the user (8) a Hght pomt (Pl) in the visual 
presentation (4a, 4b, 4c 4a, 4e) at which the pointing device (2) is aimed. 

15 3 . A method according to claim 1 or claim 2, where the chosen option is determined by 
locating a point in the template corresponding to a target point in the visual presentation 
(4a, 4b, 4c, 4d, 4e) at which the user (8) has aimed the pointing device (2). 

4. A method according to claim 3, where the light point (Pl) is located in the target area 
20 image (5) and is taken to be the target point. 

5. A method according to claim 3, where a fixed point (Pt) in the target area image (5) 
is taken to be the target point. 
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6. A method according to any of claims 1 to 5, where a desired option is selected by the 
user (8) by aiming the pointing device (2) at the desired option in the visual 
presentation (4a, 4^, 4c, 4d, 4e) and pressing a button (1 1) on the pointing device (2). 

7. A method according to any of claims 1 to 6, where the desired option is selected by 
iJie user (8) by moving tiie pointing device (2) over the visual presentation (4a, 4b, 4c, 
4d, 4e) in a pre-defined pattern. 

8. A method according to any of claims 1 to 7, where the target point is determined 
using computer vision ^gorithms. 

9. A method of any of claims 1 to 8, where the target point is determined by a method 
comprising the following steps: 

- detecting distinctive points in the target image (5) of the visual presentation 
(4a, 4b, 4c 4d, 4e); 

- determining corresponding points in the template of the visual presentation 
(4a, 4b, 4c, 4d, 4e); 

- developing a transformation for mapping the points in the target image (5) to 
the corresponding points in the template; 

- using the transformation to determine the position and aspect of the pointing 
device (2) relative to the visual presentation (4a, 4b, 4c, 4d, 4e); 

- locating the intersection point of a certain axis of the pointing device (2) 
with the visual presentation (4a, 4b, 4c 4d, 4e). 

10. A method according to any of claims 1 to 9, where the visual presentation of the 
device options (4b, 4c 4e) is presented in static form. 

11. A method according to any of claims 1 to 10, where the visual presentation of the 
device options (4a, 4d) is presented dynamically. 
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12. A method according to any one of claims 1 to 1 1, where one or more target area 
images (5) of user options for a plurality of devices to be controlled (la, U, Ic) are 
generated and compared to pre-defined templates and, depending on the option chosen, 

5 one or more of the plurality of devices (la, lb, U) are controlled accordingly. 

13. A user interface fot control of a device (la, lb, Ic), said user interfece comprising: 

- an accessing unit (12) for accessing pre-defined templates associated with 
visual presentations of user options for the device to be controlled (la, lb, 

10 Ic); 

- a pointing device (2) for aiming at a desired option in a visual presentation 
(4^, 4b, 4c, 4d, 4e) of the user options, comprising a camera (3) for generating 
an image (5) of a target area (6) of at least part of the visual presentation (4a, 
4b, 4c, 4d, 4e); 

15 - an image interpreter (13) for locating the target area (6) or a point of the 

target area (6) in a pre-defined tenaplate in order to determine the chosen 
option. 

14. A user interfece according to claim 13, comprising a transmission interface (14) for 
20 transmitting the images (5) to a control unit (1 6) assigned to a device (la, lb, Ic); 

15. A user interface according to claim 13 or 14, comprising a display unit (15) for 
dynamically displaying a visual presentation (4d) of tiie user options for the device to be 
controlled (la, lb, Ic)- 

25 

16. A user interface according to any of claims 13 to 15, comprising a hardcopy output 
unil/module for generating a static visual presentation of the user options for the device 
to be controlled (la, lb, Ic). 
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17. A pointing device (2) for a user interface according to one of claims 13 to 16, 
containing a camera (3) for generating an image (5) of a target area (6) in the direction 
(D) in which the pointing device (2) is aimed. 

1 8. A pointing device (2) according to claim 17, comprising a li^t source (7) for 
illuminating the target area (6) at which the pointing device (2) is aimed. 

19. A pointing device (2), extending along a longitudinal axis, containing a camera (3) 
positioned in the pointing device (2) such that the camera (3) generates an image (5) of 
a target area (6) in front of the pointing device (2) in the direction (D), along the 
longitudinal axis of the pointing device (2), in which the pointing device (2) is aimed. 

20. A control unit (16) comprising a receiver (17) for receiving target area images (5) 
from a pointing device (2), an accessing unit (12) for accessing predefined templates 
associated with visual presentations (4,, 4^, 4c, 4d, 4e) of user options for a device to be 
controlled (la, H, Ic), and an image interpreter (13) for locating the target area (5) or a 
pomt of the target area (5) in a pre-defined template in order to determine the chosen 
option. 

21. A device (la) comprising a control imit (16) according to claim 20. 
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ABSTRACT 

Method and s^rstem for control of a device. 

The invention desaibes a method for control of a device (la, lb, Ic). The method 
comprises visually presenting a number of user options for the device to be controlled 
5 (la, lb, Ic), aiming a pointing device (2) comprising a camera (3) at the visual 

presentation (4a, 4b, 4^, 4d, 4e) of the user options to choose a desired option, generating 
an image (5) of a target area (6) aimed at by the pointing device (2) and comparing the 
target area image (5) with a pre-defined template of tiie visual presentation (48, 4b, 4o, 
4d, 4e) to determine the chosen option. 
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